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ABSTRACT ' 

Hethods of achieving coiproaises between the 
classical laborat'ory experiment, and the constrictions affecting 
educational researchers condtjcting experiments to build upon 
correlational process-outcoie data are suggested. Ecological validity 
is achieved through partnership relationships between researchers and 
the teachers i«ple«enting treatments, design of treatments to include 
cltisters of r,elated behaviors rather tjhan just one, and provision of 
complete and detailed information about what to do and why. 
Simultaneously, a naturalistic analog of experimental control is 
achieved through judicious selection of research settings, and the 
capacity for causal inference is retained through collection of data 
regarding the degree of implementation and short term outcome. 
Finally, treatments are designed to be as exportable as possible. 
(Author/HV) 
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/ • ■ Abstract 



This paper suggests ways to achieve, compromises between the classical., 
laboratory experiment , and the constrictions affecting educational researchers^ 
who want to conduct experlmenfs to bulld^'upon correlational process-outcome 
data. I Ecological validity is achieved throu^gh fjgrtnershlp relationships with 
teachers implementing treatments, design of treatments to include clusters of 
related behaviors rather than just one, and provision of cornplete and detailed 
Information about what to do and why. Simultaneously, a naturalistic analog ' 
of/experl mental control is achieved through judicious selection of research 
settjngs, and the capacHy for causal inference is retained through collection 
of implementatjon and short ttrm outcome data. Fi n^ I lyi treatments aie do- ' 
signed to be as exportable as possible. . 
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^views by Rosenshlne and Furst (1973) and by Dunkln and BIddle (1974) 
concluded that research on teaching effects was beginning to Identify teaching 
procesn variables that correlated consistently with outcomes. This trend has 
continued,. prodtjcing a variety of replicated process-outcome relationships 
th9t can provide the bases for experimental studies. - 

r 

Programatic research that tests hypotheses derived from correlational 
work, identifies causal relationships, ^nd builds upon these in developing 



teacher education approaches Js needed i,f teaching Is to. become the applied 
science that it can and should be. This paper deals wi1h some of the complexities 
Involves in this enterprise, especially those relatlngto nonlinearity of, and 
context effects in, the process-outcome relationships used to generate experimental 
hypotheses. - ^ 

Most fundamentally, experimental follow up of correlational process-outcome 
findings Is needed because correlation does not prove causality, and correlational 
findings alone cannot provide a solid.basi^ for teaqher"education. Prescriptions 
based solely on correlations are not complete, intellectually satisfying, or even 
particularly rational. ' , 

One reason is the inherent ambiguity of correlational findings with respect 
to causaUty. Another is tbat even significant correlations usually are moderately 
strong at best. This means that even jf the teaching variables should prove to 
be causal, more information will be needed before diear and complete prescription 

will be possible. Should teachers be told, to manifest a particular behavior 

^ • ' ' /^f" 

at all times? Probably not. Sometimes 1.^*^*^-1! be Trre levant or unfeasible. 

_^w_J4l§||g^^Jrhey>4.e^^^ behavior whenever it is possible? 

Again, probably not. Positive correlations between a feacher behavior and 

student outcomes do not necessarily imply that the behavior should be maximized. 



and common sense suggests that maxlmizlnq Is seldom If ever appropriate. Instead, 
we seek to opt t ml ze- man I f est I ng , the behavior when It Is most appropriate and 



likely i6 have* posl;t.lve outconies, but not otherwise. Being abie to do this * 
requires much mbre ^Information than thei'knowledge that the behavior correlates 
wlth^ outcdmes. , , ^ 

Nonl inearity I ' 

These considerations follow from two basic attributes of process-outcome 

data concerning teacher effects. First, although correlation coefficients 

remain the most common sTatlstlcs usea ;l"o Indicate relationships, the real 

relationships are nonlinear.' This has been demonstrated in only. a few studies 

(Brdphy and Evertson, 1976; Soar, 1977), but there is every reason to assume 

th§t It holds general ly, .perhaps universally. Second, , support for a given 

rc!3t:onGhip varlcc frcn rtudy tc ct'jdy*^ Even for variables that ccnclctcntly 

^ ... 
correlate with outcomes, there tends to be a wide range in the strength of 

relaffonshlps In studies with positive results. Also, spmov studies usually fail 

\ ■ . • * - ' 

to SNp,£or1:^elationships seen in most of the other studies. 

Context Effecis ' , ^ ' ' 

Sometimes, the absence of positive results for a teaching variable that 

usually correlates significantly with outcomes can be attributed to Insufficient 

sampling. Insufficient variation among teachers, unusual measurement or methodology 

or other methodological causes. Often, though, negative findings are attributable 

to context effects on the frequencies or appropriateness of teacher behaviors. 

Some teacher behaviors are Irrelevant In certain teaching contexts, so that 

observation during these contexts w.ill reveal little or nothing about the effects 

of these behaviors, because they will not odcur. Also, when a teacher behavior 

♦ 

occurs In more than one context. It sometimes happens, that It Is appropriate and 

\ 
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effective In one context but not in the other (Brophy and Evertson, Note, !)• 
These contextual phenomena Indicate that generalizations must be limited ^o 
the appropriate contexts. 

These are the basic reasons that underl le the observed nonllnearlTy of 
relationships, as weTL Closer analysis of a nonlinear relatlonshl'^^ 1 1 reveal 
that the teacher behavior. Involved Is relevant and optimal In certain contexts, 
relevant but not optimal In other contexts (where some alternative Is optimal), 
anc: irrelevant in still other contexts. Measuring the behavior without attention 
to these context differences will yield curvilinear relationships to outcomes, 
but If one takes context Into account, r'elat lonsh ips that are/more specific and 
linear will, be revealed. These relationships are context-specific and thus 
not generlcto all teaching situations, but they also. can be lead to much 
more specif ic, prescriptions to teachers. 

Interpreting Correlations ^ • 

Th<^ prospect of using process-outcome correlations afe the basis for 
exper imentjs 'on teaching Is attractive, partly because It seems to be a way 
to Identify Important teaching behaviors and virtually guarantee positive 
experimental results. However, the inherent ambiguity of correlational data,, 
the fact that most relationships prove to be nonlinear when analyzed carefully, 
and the fact that most have to be qualified to take context effects Into account, 
ii«ke this task much more complex tijat It seems at first. Furthermore, confining 
Ofjese.lf to consideration of linear relationships or relationships that are 
especiajly strong will only mask these ^complexities, and attempts to get rid of 
them by using box scores to "settle" disagreements will falL Ultimately., we do 
no4 really understand process-outcome relationships until we can explain both why 
they occur when they they do occur and why they do not occur when they do not . 



There are no established parafdigms for developing this depth of understanding, 
but much progress can be made simply by observing instruction, interviewing 
tiachers, and bringing to bear information froiji other research • Analyses of ^ 
data that seem to prescribe different behavior in different situations Is 
especially valuable, because such ;dat,a provide clues to the factors that Influence ^ 
the, appropriateness and probable success of particular teacher behaviors. Also, 
3t Is instructive. to identify the alternatives to a particular behavior of interest 
w)>en trying to develop hypotheses about why the behavior is or is not effective in 
particular situations. Consideration ^f the alternative behaviors aval I atiie to 
trie teacher In these situations often leads to insights about reasons for the 
success or failure of each of the alternatives. . .. 

Consider the alternatives available to teachers when students do, not respond, 
or respond Incorrecrly, fol.lowlng a question. First,- the teacher must decide 
yhoth3r to provide the correct onevtur or lo Cwii;{ Inue \o qudbHwH nic? siuJcai Im 
an attempt to get a correct response. If the correct answer Is provided, the 
teacher can |do this personally, can call on another student, or can allow other 
students to call out the answer. If the teacher decides to address another question 
to ihe original student, this can be a repeMtlon of the original question, an 
easier question (either a newer and simpler oi^or an elaboration of the earlier 
one that prwldes clues and thus makes It easier to answer), or a. question designed 
to Identify the student*s problem rather than get an answer to the original 
question (asking whether or not the student has heard the question, has read the 
assignment, or has done preparatory work). 

'Previous research and logical analyses of these alternatives suggest that 
some would be optimal In some situations but unwise In others. For example, 
repeating the question without giving any help might make sense If the student 



has not responded, but not if the student has given a wrong answer. Also, olvlng 
.the answer probably Is preferable to -calling on other students or letting them 
caM out answers, but hot necessarily preferable to asking the student another 
question In an attempt to elicit a response. Giving the answer probably is most 
appropriate when. the teacher is introducing new material or when the student 
•seems completely confused^ while attempting to elicit a response through repeated 
questioning feeems more appropriate when there Is reason to believe that the student 
knows the answer .or could figure It out with help. 

Deyelopin'q Hypotheses ^ 

^Distinctions like these elaborate knowledge about the situational differences 

« ♦ ^ * 

that InfJuenc6 (or should influence) teacher behavior, and they place teacher 
behavior^ Ir* contexts that specify antecedent conditions and short run. outcomes 
rA|?j+ori -f^^ goa's The r«»s*j!+ fs the developrent of 2 set of .related hypothe*:es 
.c6ncerning\when particular teacher behaviors are optimal and why they succped . 
when the/ do,^ Taken together, such hypotheses provide systematic expjan^tlons 
for process-outcome relationships, and they allow specific and differential 
predictions about what will happen If particular alternatives are used in partlcula 

\ • ■ • • 

situations. T ' . 

This makes it possible to train teachers specifically and exhaustively, and 

to test sophisticated differential hypotheses relating alternative behavior to 

' . ' - - \ ' 

short term and long term outcomes. | 



Treatments as iC lusters of Hypotheses '^^ ^ 

'An Important implication here Is that teaching behaviors manipulated in 
experiments on teaching usually should be considered In related ciusters rather 
than In Isolation. The classical experiment Involves systematic manipulation of 



a single Independent variable combined with precise control over. all other variabi 
known to.be re levants Even If this were feasible in e^ducation, and it seldom is, 
it probably would not be appropriate. We have 1o sacrifice this kind of precision 
and control In. order to achieve ecological validity (which exists when the 

i . ' i - \ 

experimental .situation is neair enough to naturalistic teaching i-p allow us to . . 
general l2e experimental results to the classroom). Also, given the generally 
primitive development of oal^come measures and the difficulty of measuring change. 



it often Is necessary to manlpujate a cluster of related teaching variables 



. simultaneously In order to create trea:hnents that are powerful enough to have 
detectable effects. • t 

This need not be a flaw In the experimental (design or an Impediment to 
_^_jcausa-H:lnference if experiments are designed so that Implementation data and 
Information about short t erm outcomes are col lected.\ Implementation data 



ai Treatment is being impiemenreo by 



' i 

conct^ni the degree ro wnicii ine experiiDenii 
the/teaqhers In experimental groups, and < Iso the degree to which the treatment 
is being "Implemented" natural istlcal ly by the teachers in control groups. 
This information is. Important In. educational research, because complete control 
over teaching behavior usually Is neither possible nor desirable. 

Complete control* usual ly Is not possible because treatment Implementation 
depends upon the experimental teachers, who may forget or be unable to behave 
as directed at all times or may deliberately substitute for the treatment at 
times whenjt seems Inappropriate. Even when full control of the treatments 
used In each group lsT)ossfble, It usually Is not d^lrable because it creates 
artificial situations In all groups. The practical questions of interest In 
educational research concern comparisons, of proposed Improvements with common 
practice, not comparisons of hypothesized ideal treatments with their opposltes. 
Treatments based on clusters of related behaviors rather than /Isolated single 



behaviors hefp facilitate this in two ways. First, they Jncrease the chance that 
effects will be practically significant as' we! I as statistical Jy significant. 
Second, they are more acceptable intel loctual ly and emotional Jy to teachers, and 
for that reason are easjer to implement consistently, than treatments confined 
to a single behavior. 

ImplementatLofV-and-^fiort term outcome data allow one to test predictions 
concerning each separate behavior involved in the treatment. If tne contexts 
In'whlch each behavior Is relevant arc clearly spelled out, and If the short term 
oul^omes of performance of the desired behavior are predicted (improved attention, ^ 
Improved response, ability to complete a task .with out seeking h^lp, etc.), 
teachers can be coded for lmplemen"*"atlon of the treatment in the relevant contexts, 
and the short term outcomes following successful or unsuccessful implementation 
can be coded as well. If there is sufyclent variation in teacher behavior, 
(cnovon !n:p!2n:cntat!cn in the, trcatr.cr.t group nr.d/o^ some naturallctlc ir.plc.v.cntct!c 
in the control group), l;f.wl|l be possible to determine whether or not the behavior 
In question was systematical ly associatec^ with long term outcome measures. 



Again, this can be done for both the troatmei^h and the control groups, assuming 

sufficient variance.* \ I 

' - , ^ " \ \-\ ' ' " I 

These design features strike a compromise between the controlled laboratory 

experiment and the need for ecological validity to provide a basis for generalizing 

the findings. The Interrelationships among the treatment elements and the 

rationales that underlie and tie them together make the treatment both more powerful 

'and nore face valid for teachers, yet the design allows for specific evaluation 

of each treatment element In addition to assessment of the effectiveness of the 

treatment as a whole. 

■ . . ^ \. 
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f * / Chotce of Research Settings * * 

/ ^ it is Important to find a research setting that will provicje an appropriate, 

and If possible optimal', opportunity to test the. hypotheses involved. Sometimes 
^ the hypotheses point to a particular setting, as when they deal with small group 

instruction or Instruction In particular subject matter at a particular grade 
level* Even where no such specification exists, however^ and even where the 
^ treatment is bel ieved togeneral ize toal I "teaching," some, settings wl 1 1 be better 
\ than others. This Is because certain assumptions are built into treatment 

hypotheses even if the investigators are not aware of them. ^ 

One of these* concerns the classroom atmosphere. ^ Jnless ihe treatment deals 
vi, witn classroom. management, hypotheses relating teacher behavior to outcomes 
- probably assume'^ motivated ^ or at least cooperative students who pay attention to | 



ERLC 



Instruction and apply thems^Jves In working on assignments^ Where this Is not ; 

An appropriate match .between the objectives arjd materials of the curriculum 
and the present abilities and interests of the students usually is assumed, as 
w^lU if the tasks involved In the' treatment are. much too easy, celflng effects 
will minimize variance in outcomes, and thus fr.ake it difficult or impossible to 
test the f)ypotheses. Alternatively, If the tasks are much too difficult, even 
well implemented treatments that might have been s.uccessful under other conditions 
will not yield enough learning to make a detectabJe difference. This danger Is 
qui^e real whenever "grade level" tasks are used in schools or classrooms where 
most of the students are functioning at level s' slgnificantl y below grade level. 

There usually are implicit assumptions about teachers, as well. Typically, 
successful implementation means not only that the teachers perform the experimental 
behaviors as instructed, but also that they be generally competent in handling the 

■ 

■ 12 ■ 



broad range of tasks that teachers giust able to do. At mfnfmum, this would 
seem to require experienced teachers, jand perhaps teachers experienced with the 
particular grade levels and curricula Involved. Even among these teachers. It may 
be necessary to exclude those who are Inept as classroom managers or otherwise 
deficient In ways that cause their* classrooms to differ fundamentally from most * 



others and preclude meaningful tests of hypotheses about instructional techniques. 

An imp! Ipation of ihese conslderatjons about research context is that experiments 
doslgncd foulest causal hypotheses based on process-outcome correlations should 
be conducted in ideal settings , at least . Initial ly. The teachers should be 

■/ . 1 . ■ • / 

experier)ced, competent, and committed yo implementation of the treatment, and the 
students should be appropriate' In demographic characteristics, matched to the 
content and difficulty level of the curriculum, and socialized to the student 
role. This will have the effect of maximizing the chances for positive results, 
wttfc!i i^ do II oliwulJ bte Wiitsti liic iebiUtg of hypuihteses aboui ihe. i r tjalmeni . 13 
the primary interest. In effect, selecting the Ideal setting In whiQh to conduct 
such research is a way to control for other factors that might Interfere with 
treatment implementation or reduce the variance In learning outcomes ^o a point 
where' hypothesis testing would be Impossible. 

Chaotic settings should be avoided In early experimental work designed to 
estabi I'Sh^reatment effectiveness. Later, after the effectiveness of the treatment 
has been established In Ideal settings, attention can be focused on the classrooms 
of student teachers, teachers who cannot control the class, teachers who must 
deal with drastic variations In student ability and achievement levels, teachers 
tacing entire classrooms full of alienated low achievers, and other classrooms 
where there may be less chance for the treatment to work. 



Training tHe Teachers . 



• • In, expefTimental^ l^stfe^of teaching principles, the teachers should bj cooperating 

• ' ^ . ^ ' '^N^ * ' ' ' - 

partners with the experlmenterr7*o±J[|subjccts" given minimal Information. Researchers 
concerned about Hawthorne effects and experimenter bias effects sometimes give 

. • ■ / • -I' - 

t-eachers minimal information about what to ^o and/or why to do It. This approach 
probably does more harm than good. It reduces cooperation ar$d good will, confuses 
the teachers, minimizes Implementation, and Introduces artificiality. This works 
against the goal of getting teachers to Implement the treatment both natural Istlcal ly 
and wel I ^ in order to provl.de a credlMe^4^as^s for generalization.' 

A better way to avoid Hawthorn^ and experimenter bias effects Is to have two 
or more experimental groups ir^iol^ent different treatments.. If specif Ic differential 
hypotheses, a re deve,loped an^short term (^tc^me data are collected, it wl 1 1 be 
possible to see If each group implements''* its treatment successfully and gets 
sys'torr^ttca! ly ^ood rosultc on outcomes related to Its trectment* Ihxz design 
\ not only equalizes Hawthorne effects a(id experimenter bias effects, but it 
\proyldes good a'ssurance that the treatment effects of interest to one group are 

\ ■ ■ '■ \ ■ 

not of interest to the other. ' * ( > 

\' ' " < 

\ Whether or not there Is more than one experimental group, it Is possible 

to. have control groups that do not know about the treatment groups and can be 

mor^ltored for naturalistic Implementation of treatments. Teachers in these 

\" • 

groups can be motivated to cooperate in much the same way that they would be 
mot-^i^ated if Involved In a purely naturalistic process-outcome study. They would 
be piptlcipat Ing In research and getting just as much attention from the 
experimenter, but they would not be asked to Implement specific treatments and 
thus would not have thelf attention called to the /behaviors of Interest. The 



possibility pf ex[>ej>ftfienter bias effects could be assessed by analyzing the short 
term outcomes in these control classrooms and those of the experimental classrooms. 
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ThTs would Involve comparing the Intensity, enthusiasm, and other qualitative 

aspects of behavior observed when situations relevant to the treatment principles 

- f^' • . ' ' m ' ' ' ^ ' 

ards^r, .and, .more Importantly, comparing the short term outcomes of ihe behavior 

Of these two teacher groups when they did and did not Implement the treatment. 

Treatments themselves should oe face vaJid and well articulated. In 

training the teachers. It is Important to circumscribe the contexts for decision 

' ' - . . i * , 

and action, identify relevant and approved" strategies for each context, specify 
alternative strategies if more than one way to operate is available, and explain' 
about when and why each alternative is best or not advised. Context information 
that tells when a strategy is relevant may be Just as important as explanations 
about what the strategy involves^^and how it should be Implemented. Teachers' 
abilities to recognize contexts that are appropriate and remember the relevant t 

» ' *§* 

strategies will facilitate their ability to implement the treatment properly. 

.^-^^ 

if sequences oTccjr.pl 3K behavior crc involved, the .treatment should Include 
the whole sequence, complete with alternative behaviors relevajTl^^at^^li fferent 
stages. Instructions should cover" except ions to the general' rules, along with 
1;he rules themselves. Again, it is esse'ntially Irrational to present teachers ^ 
With partial Information and asif them to implement certain behaviors at all times 
when it really is recommdnded for certain (unspecified) times ("Be indirett," 
"Ask complex questions"). Teachers should know the general principles behind 
/prescriptions, not just the prescriptions themselves, so that they can adjust 
appropriately to situations that the prescriptions do not cover. 

' Short Term Outcomes * ' . ^ • 

The importance of short term outcomes must be stressed. To monitor treatment 
Impjementation, It will be necessary to develop ways to convert behavioral objectives 
into percentage scores reflecting the implementation of decision rules (as oppbsed 



to simple frOTuency or rate measures that give Information about how often a 
behavior waslobserved but do not say whether its appearances were appropriate 
or not). ^ ^ I t 

Information about the. linkages between treatments and short term /Dutcomes 

' ^ ' ■ ■ /' J 

tallows Inferences about how t real, nents work in addition to documenting that they, 

dn work^ This Js useful whether or no+^ infnrma+inn aboii+ Innn +erm ynii+mmos / 



do^^work. This Is useful wtiether'or not^lnformatlon about long term ybutcomes 

Is available. In fact,. It fs more prescriptive and sclentif Ica^l ly.4onvlnci,ng/ 

} ' * / f i 

fhjn data relating to long term ^outcomes./ We probably have pigced too much/ 



-emphasis on long term outcon)es, especially the kinds of omnibus achievement/ 
tests that provide a credible global Indication of how students are progressing 
but do not allow specific linkages between their progress and anything that the 
teach'er ^does^^ ^ o uch outcomes may have been overemphasized to the point that 

wort hwhl I e Information about teaching has been Ignored because it has not been 

> ' ^ ^ i ' ■ ..." . r ' ^ 

supported ,<jvith ;cng term outccjno data. . ^ ^ . . 

We should. retain our concern with -outcomes, especially achievement outcomes, 

as the criteria ^r effective teaching, but 'we need to improve upon bur use of 

omnibus achievement tests. We need more iJelineation of specific differential ' 

outcomes hypothesized to. result from speclTIc differential teaching behavior,, 

along with tests of hypotheses through well deslgired experiments that yield 

credrble^ data. Sometimes, differential predictions will ha^'e more to say 

about the quality or type of achievement than the amount, but this is as It 

should be. Before decision makers can rationally evaluate alternative teacher ^ 

behaviors, it will be necessary to document systematic relationships between / 

these teacher behaviorsfand their contrasting outcomes. Most of this will jnvo/ve 

the tise of short term outcome data. 
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reatment Exportabl I Ity 



.. Although It Is Important to make the treatment as complete and credible as » 

>o&&Jbie in the sense described above, it also Is Important to hold the duration, 

Lxpense, and general intensity of the treatment to a minimum. , This Is recommended 

not so much for scientific reasons as for practical reaspns relating to treatment 

exportabl I Ity. The more thorough the treatment, the more likeiy systematic 

implementation, but the less general izable to ordinary teaching. 

In view of this, it probably is best to start, where feasible, with minimal 

treatments that Invojve 1 1 ttle external help In the form of supervision, video- 

taped feedback, or other time consuming or. expensive features. .If such treatments 

work (Implementation is good), and If they produce desirable effects, they can be 
' , ' - N J. • _ , 

exported quickly and ch^iply. If. not, more high powered treatments can be 

tried out .in subsequent work (although the problem m^y be lack of specificity, 

viailly, wi oven ut-^rrec{rie:35) of Trie irvdirrieur itseii raitfCjt liian difficulhy 

• - -* < ^ • ^ - - " • 

of getting teachers to master, it or to Implement It). 



' Summary^ 

This paper has suggested ways to achieve compromises between the classical 
laboratory experiment and the constrictions affecting educa'fional researchers 
who want to conduct experiments to build upon cortelational process-outcome data* 
Ecological validity is achieved through partnership relationships with teachers 
Implementing treatments, design of treatments to Jnclude clust^ 
behaviors rather than just one, and provision of complete and detailed information 
jabcut what to do and why* Simultaneously, a naturalistic analogue of experimental 

control Is achieved through judicious selection of research settings, and the 

. " • ' 1 

capacity for causal inference is retained through collect ion .pf Implementation and 

short term outcome data. Finally, treatments are designed to ,be as exportable 

. ; ^ 

as possible. 

17 ' ; , 
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